The present invention relates to machine learning. In particular, the present invention relates to reducing training time for machine learning.
Machine learning is a general term that describes automatically setting the parameters of a system so that the system operates better. One common use for machine learning is the training of parameters for a system that predicts the behavior of objects or the relationship between objects. An example of such a system is a language model used to predict the likelihood of a sequence of words in a language.
One problem with current machine learning is that it can require a great deal of time to train a single system. For example, it can take up to three weeks to train some language models. This problem is especially acute in systems that have a large number of parameters that need to be trained and a large number of outputs, and whose training time is proportional to the number of parameters times the number of outputs of the system. In particular, systems that utilize Maximum Entropy techniques to describe the probability of some event tend to have long training times.
Thus, a method is needed that accelerates training time for systems that have a large number of outputs.
A method and apparatus are provided that reduce the training time associated with machine learning systems whose training time is proportional to the number of outputs being trained. Under embodiments of the invention, the number of outputs to be trained is reduced by dividing the objects to be modeled into classes. This produces at least two sets of model parameters. At least one set describes some aspect of the classes given some context, and at least one other set of parameters describes some aspect of the objects given a class and the context. Thus, instead of training a system with a very large number of outputs, the present invention trains at least two smaller systems, at least one for the classes and one for the objects within a class. Since the number of outputs of each of the new systems may be as small as the square-root of the number of outputs of the original system, the resulting training times may be considerably reduced.
Under many embodiments of the invention, maximum entropy models are trained by training a first set of maximum entropy weighting values and a second set of maximum entropy weighting values. The first set of weighting values is used to determine the probability of a class given a preceding sequence of words. The second set of weighting values is used to determine the probability of a next word given a class and a preceding sequence of words.